Skip to content

Implement an M3 reporter - #12

Merged
justinjc merged 44 commits into
masterfrom
m3-reporter
Oct 24, 2017
Merged

justinjc merged 44 commits into
masterfrom
m3-reporter

Conversation

@justinjc

Copy link
Copy Markdown
Collaborator

Implement an M3 reporter. Additional tests will come in a separate PR to reduce PR size.

@coveralls

coveralls commented Sep 12, 2017 •

Copy link
Copy Markdown

Coverage Status

Coverage decreased (-20.3%) to 40.444% when pulling f7186dd on m3-reporter into 4879992 on master.

Comment thread ci/before_install.sh Outdated
echo "security.provider.1=org.bouncycastle.jce.provider.BouncyCastleProvider" | sudo tee -a /etc/java-7-openjdk/security/java.security

# Install Thrift 0.9.3
sudo apt-get update -qq

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do you need to install thrift binaries? if only for the codegen, might be better to include the generated code in the diff.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two options here:

  1. Yes, installing thrift binaries here are for codegen, but since we compile thrift files as part of the project build, we are also allowing Travis to test our build process as a developer would build the project. However, installing thrift binaries are by far the most time consuming part of a Travis build.
  2. The other option is that we just check in the generated code, which means we'd have to specify Travis-specific build parameters to not have it try to compile thrift.

Thoughts?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think developers are going to be regening thrift code often at all, so (1) seems excessive. I suggest (2).

@justinjc justinjc Sep 13, 2017 •

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've gone ahead and made Thrift code generation skipped by default in addition to checking in the generated source code. Code generation can be triggered by passing a property (documented in DEVELOPER.md).

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Solid

@@ -50,15 +48,10 @@ class ScopeImpl implements Scope {
// performance upside to the consequence of reporting a newly-made metric in

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove/update the comment to reflect synchronized usage?


static class Values {
private ReadWriteLock rwlock = new ReentrantReadWriteLock();
private final ReadWriteLock rwlock = new ReentrantReadWriteLock();

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why use a rwlock here v synchronise in other places?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

synchronized is used when I would only use one (write) lock. We're making use of ConcurrentHashMap's ability to read values in a thread-safe manner without locks. This rwlock is to protect ArrayList accesses (there isn't a similar Concurrent* version) and having separate read/write locks allow for higher throughput.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

makes sense, could you put a note in the code explaining that intent

executor = builder.executor;

addAndRunProcessor();
} catch (Exception e) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This try-catch is going to loose Exception type information. E.g. it'll make IOExceptions (e.g. Thrift connection issues) into IllegalArgumentException. Can you avoid doing that?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because I hand the Exception e over to my throw, the exception call stack will be preserved, e.g. it'll say IllegalArgumentException "Exception creating M3Reporter" caused by SocketException "something else".

I agree this isn't the prettiest, but an exception here probably signifies an IllegalArgumentException e.g. an incorrect/already used port. We also need to convert these checked exceptions to unchecked exceptions because we can't recover and proceed from these errors.

protocolFactory = new TBinaryProtocol.Factory();
}

TTransport transport = new TUdpClient(hostPorts[0]);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why have hostPorts be an array if you're going to use only one?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was done in anticipation of supporting having a transport support multiple host/ports together like the Go version. Listed as issue #11.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool, sounds good.

@Override
public void run() {
try {
List<Metric> metrics = new ArrayList<>(freeBytes / 10);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how'd you arrive at freeBytes/10?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

per our chat yesterday, seems like this number is just straight from the go implementation.

Could you add an issue to profile the code for different workloads, to gauge expected perf, and to size the defaults/constants correctly. Might be worth applying for YourKit's free OSS license

return metric;
}

private Set<MetricTag> addMetricTagsToSet(Map<String, String> tags) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: rename to indicate you're creating a set instead of adding to a provided set (e.g. toMetricTagSet)

* @param protocol the value to set
* @return this {@link Builder} with the new value set
*/
public Builder protocol(ThriftProtocol protocol) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will users ever override this?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, they can specify ThriftProtocol.BINARY as opposed to the default TBinaryProtocol.COMPACT, which will make the reporter use that TProtocol accordingly.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah but will users ever have to set this to the non-default value? I'd rather not reveal the transport mechanism as a choice to the users if we can help it.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems like TCompactProtocol is the more efficient protocol here, so I'll just default to using that.

TimerValue timerValue = resourcePool.getTimerValue();
timerValue.setI64Value(iValue);
metricCopy.getMetricValue().setTimer(timerValue);
break;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

exception for default?

@@ -189,17 +194,23 @@ public int compareTo(Duration other) {
public String toString() {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you put a javadoc explaining the expected output

} else if (nanos == Long.MIN_VALUE) {
// Return hard coded response as workaround because
// Math.abs(Long.MIN_VALUE) == Long.MIN_VALUE
return "-2562047h47m16.854775808s";

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this feels hacky, why do you need the special case?


StringBuilder buf = new StringBuilder(16);
// 25 capacity from max String length produced from Long.MIN_VALUE nanos
StringBuilder buf = new StringBuilder(25);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you make a static final for 25


private final MetricBatch metricBatch = new MetricBatch();

private final Object calcLock = new Object();

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just occurred to me, can't you use the object itself for locking? as in why have an additional member called calcLock when you can synchronise on calc

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

calcLock protects both calc and calcProtocol. I'll put in a comment.

return;
}

TException exception = null;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

considering you're just logging, does this need to be an exception still?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nope, will remove and just log.

int tries = 0;
boolean tryFlush = true;

synchronized (metricBatch) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does making metricBatch an instance member instead of alloced/cleaned up in the flush call buy you anything?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. Made metricBatch a local variable and add commonTags to it instead - as discussed offline, should help with throughput as this removes the synchronized block.

// Holds metrics to be flushed
private final BlockingQueue<SizedMetric> metricQueue;
// The service used to execute metric flushes
private ExecutorService executor = Executors.newFixedThreadPool(THREAD_POOL_SIZE);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: make this configurable in the builder.

public void run() {
try {
while (!executor.isShutdown()) {
// This `take` call will block until there is an item on the queue to take.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: update this comment to indicate why you're using poll instead of take

}
}
} catch (InterruptedException e) {
Thread.currentThread().interrupt();

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry I'm missing something. A processor is already running on a different thread, and it's received an interrupt, why does it need to interrupt itself again?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops good catch - missed this earlier.

// catch block.
SizedMetric sizedMetric = metricQueue.poll(MAX_PROCESSOR_WAIT_UNTIL_FLUSH_MILLIS, TimeUnit.MILLISECONDS);

if (sizedMetric != null) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you restructure for readability:

if (sizedMetric != null) {
  flushMetricInteration(sizedMetric);
  continue;
}

// If we don't get any new metrics after waiting the specified time,
// flush what we have so far.
if (!pendingMetrics.isEmtpy()) {
  flushMetricIteration(SizedMetric.FLUSH);
}


// Important to use `shutdownNow` instead of `shutdown` to interrupt processor
// thread(s) or else they will block forever
executor.shutdownNow();

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think you need to re-order: i.e. isOpen = false; before this, no?


@Override
public void reportCounter(String name, Map<String, String> tags, long value) {
synchronized (isOpenLock) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this (synchronized on isOpenLock) affect throughput much?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As discussed offline, removed the need for an isOpen flag. Instead, use a sentinel metric in the queue to indicate that all metrics coming after it was put in after reporter was closed.

BucketPair[] bucketPairs = BucketPairImpl.bucketPairs(buckets);

if (tags == null) {
tags = new HashMap<>(2, 1);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what does the (2,1) here mean?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed this optimization for readability.

}
}

private class Processor implements Runnable {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you break this class out into a separate file? feels like there's too much going on in this file

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After attempting this, I found that the Processor is too tightly integrated to the M3Reporter to meaningfully have it broken out of it. The shared metrics queue, flushing logic, and synchronization variables will probably not enable code reuse if this was split up.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair enough


if (tries < maxTries) {
try {
Thread.sleep(FLUSH_RETRY_DELAY_MILLIS);

@prateek prateek Oct 18, 2017 •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

try catch in a try catch feels like smell, can you that clean up

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed that this isn't the prettiest code, but I want to sleep only when I hit an exception, and sleep throws a checked exception 😕. Extracting out just a sleep function seems a little silly.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As discussed offline, there seems to be little upside to having this retry logic in the first place - removing this for now.

BucketPair[] bucketPairs = BucketPairImpl.bucketPairs(buckets);

if (tags == null) {
tags = new HashMap<>(2, 1);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use const vars rather than literals heres

@justinjc justinjc Oct 18, 2017 •

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These numbers are used for their specific circumstances. If tags == null, then we know that there will only be two items in the map (the two that we add later on). The 2 is the initial capacity of the buckets, and the 1 defines when to rehash.

Added a comment explaining this.

return;
}

boolean tryFlush = true;

@prateek prateek Oct 24, 2017 •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: i don't think you need this var or the while loop wrapping the try below anymore

// When this reporter is closed, shutdownNow will be called on the executor,
// which will interrupt this thread and proceed to the `InterruptedException`
// catch block.
SizedMetric sizedMetric = metricQueue.poll(MAX_PROCESSOR_WAIT_UNTIL_FLUSH_MILLIS, TimeUnit.MILLISECONDS);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you make this configurable, and change the default to 10s (default reporting interval in go-tally iirc)


if (sizedMetric == SizedMetric.FLUSH) {
flush(pendingMetrics, commonTags);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: remove newline here

int size = sizedMetric.getSize();

if (metricsSize + size > freeBytes) {
flush(pendingMetrics, commonTags);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A question for you ~ what happens if we get a lot of timer values emitted in a short burst. E.g. the current java client runs into this with metrics for a Kafka topic being processed.

I'm afraid of this - we'd queue all the timers directly into the metricsQueue, because there's no buffering. If the processor thread doesn't get scheduled during the queuing, we'd fill the queue, and be unable to flush because we can't queue the sentinel flush metric. Can this happen?

@prateek prateek Oct 24, 2017 •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

talked offline, not a deadlock because we can still make forward progress despite not being able to enqueue a flush. We'd drop metrics if the queue was full but that's an accepted semantic already.

@prateek prateek left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@justinjc
justinjc merged commit 092fdb0 into master Oct 24, 2017
@justinjc
justinjc deleted the m3-reporter branch October 24, 2017 20:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants